Apparatus and method for multispectral imaging with parallax correction

ABSTRACT

An apparatus and a method for multispectral imaging that may, comprising a cyclopean camera arrangement comprising at least two spaced apart imaging sensors operating at a first waveband, an image combiner arranged to combine images from the at least two spaced apart imaging sensors to provide a first cyclopean image of a scene from a first virtual position, a further imaging sensor operating at a second waveband different from the first and arranged to provide a second image of the scene from a second position, and a second image combiner arranged to combine the first and second images of the scene to provide a multispectral image, wherein the first virtual position of the cyclopean camera arrangement and the second position of the further sensor are arranged to be substantially the same.

PRIORITY CLAIM

This application claims the benefit of U.S. Provisional Application No.61/791,515, filed Mar. 15, 2013, the content of which is herebyincorporated by reference in its entirety.

TECHNICAL FIELD

The present invention relates to apparatus and methods for multispectralimaging. Certain embodiments relate to methods and apparatus combiningvisible light and infrared images of a scene to provide a combinedmultispectral image that has been corrected for parallax. Certainembodiments comprise the three dimensional overlaying of visible lightand infrared images of a scene, and in particular apparatus and methodsfor enhancing the visibility of the displayed thermal information ofimaged objects within the scene.

BACKGROUND

Infrared imagery of a scene can be used to provide information notobtainable from visible light imagery, such as temperature informationnot discernible in visible light imagery.

However, infrared imaging systems frequently operate at lower levels ofspatial image resolution to that provided by visible light imagingsystems.

To provide a more informative image, infrared and visible light imagescan be overlaid in order to obtain a resultant combined image. Thevisible light imagery content can be used to provide clearlyrecognizable details relating to objects within the scene, and theinfrared imagery content can be used to provide further information,such as for example temperature information relating to these objects.

However a problem may be caused because the visible light and infrared(IR) cameras are located at different spatial positions when viewing thescene. As a result the visible and IR cameras will see objects in thescene from different angular positions. This can lead to parallaxeffects which may cause problems.

Furthermore, the imaged scene can contain a total number of objects witha wide range of temperatures, while an object or objects of interestwithin the imaged scene may have temperatures that extend over a smallertemperature range. Within the combined visible light and infrared image,or indeed within an infrared image, the effectively displayable contrastacross the object or objects of interest is reduced because thedisplayable contrast range of the infrared display must extend over theentire temperature range of the total number of objects within theimaged scene.

There is furthermore a need to extract as much information from combinedvisible light and infrared images of a scene, which can be difficultusing existing imaging systems where spatial information within thescene may not be discernible or be lacking within captured images.

It is an aim of the present invention to solve these problems.

SUMMARY

In a first aspect the invention provides an apparatus for multispectralimaging, comprising:

a cyclopean camera arrangement comprising at least two spaced apartimaging sensors operating at a first waveband;

an image combiner arranged to combine images from the at least twospaced apart imaging sensors to provide a first cyclopean image of ascene from a first virtual position;

a further imaging sensor operating at a second waveband different fromthe first and arranged to provide a second image of the scene from asecond position; and

a second image combiner arranged to combine the first and second imagesof the scene to provide a multispectral image;

wherein the first virtual position of the cyclopean camera arrangementand the second position of the further sensor are arranged to besubstantially the same.

In certain embodiments, the second image combiner is the image combiner.

In certain embodiments, the first waveband is visible light and thesecond waveband is infrared.

In certain embodiments, the first waveband is infrared and the secondwaveband is visible light.

In certain embodiments, the further imaging sensor comprises an imagesensor element, wherein the first image from the cyclopean cameraarrangement appears to be generated on a virtual image sensor elementwithin a virtual camera, wherein the image sensor element of the furtherimaging sensor is positioned at the spatial position of the virtualimage sensor element of the virtual camera.

In certain embodiments, the further imaging sensor is located betweenthe two spaced apart imaging sensors.

In certain embodiments, the further imaging sensor is located at anequal distance from each of the two spaced apart imaging sensors.

In certain embodiments, the cyclopean camera arrangement comprises twospaced apart imaging sensors housed within separate housings.

In certain embodiments, the further imaging sensor is separate from eachof the housings for the two spaced apart imaging sensors.

In certain embodiments, the further imaging sensor and the two spacedapart imaging sensors are all comprised within a common housing.

In certain embodiments, the first image and second image are provided tothe second image combiner at substantially the same level ofmagnification.

In certain embodiments, the second waveband is at least one of: thethermal infrared wavelength range, the mid infrared wavelength range,the near infrared wavelength range, and the long infrared wavelengthrange.

In certain embodiments, the first waveband is at least one of: thethermal infrared wavelength range, the mid infrared wavelength range,the near infrared wavelength range, and the long infrared wavelengthrange.

In certain embodiments, the at least two spaced apart imaging sensorsare cameras and/or the further imaging sensor is a camera.

In a second aspect the invention provides a method for multispectralimaging, comprising:

obtaining a first cyclopean image of a scene from a first virtualposition using a cyclopean camera arrangement comprising at least twospaced apart imaging sensors operating at a first waveband by combiningimages from the at least two spaced apart imaging sensors to provide thefirst image of the scene;

obtaining a second image of a scene from a second position using afurther imaging sensor operating at a second waveband different from thefirst; and

combining the first and second images of the scene to provide amultispectral image;

wherein the first virtual position of the cyclopean camera arrangementand the second position of the further sensor are arranged to besubstantially the same.

In certain embodiments, the first waveband is visible light and thesecond waveband is infrared.

In certain embodiments, the first waveband is infrared and the secondwaveband is visible light.

In certain embodiments, the further imaging sensor comprises an imagesensor element, wherein the first image from the cyclopean cameraarrangement appears to be generated on a virtual image sensor elementwithin a virtual camera, wherein the image sensor element of the furtherimaging sensor is positioned at the spatial position of the virtualimage sensor element of the virtual camera.

In certain embodiments, the further imaging sensor is located betweenthe two spaced apart imaging sensors.

In certain embodiments, the further imaging sensor is located at anequal distance from each of the two spaced apart imaging sensors.

In certain embodiments, the cyclopean camera arrangement comprises twospaced apart imaging sensors housed within separate housings.

In certain embodiments, the further imaging sensor is separate from eachof the housings for the two spaced apart imaging sensors.

In certain embodiments, the further imaging sensor and the two spacedapart imaging sensors are all comprised within a common housing.

In certain embodiments, the first image and second image are provided tothe second image combiner at substantially the same level ofmagnification.

In certain embodiments, the second waveband is at least one of: thethermal infrared wavelength range, the mid infrared wavelength range,the near infrared wavelength range, and the long infrared wavelengthrange.

In certain embodiments, the first waveband is at least one of: thethermal infrared wavelength range, the mid infrared wavelength range,the near infrared wavelength range, and the long infrared wavelengthrange.

In certain embodiments, the at least two spaced apart imaging sensorsare cameras and/or the further imaging sensor is a camera.

In a third aspect the invention provides apparatus for imaging,comprising:

representation generating means arranged to generate a three dimensionalrepresentation of a scene;

at least one infrared imaging sensor arranged to obtain an infraredimage of the scene; and

overlaying means arranged to overlay the infrared image onto the threedimensional representation of the scene to produce an infrared threedimensional representation of the scene.

In certain embodiments, the three dimensional representation of thescene is a three dimensional model.

In certain embodiments, the apparatus further comprises at least onevisible light imaging sensor arranged to obtain a visible light image ofthe scene; and

wherein the overlaying means is arranged to overlay the infrared imageand the visible light image onto the three dimensional representation ofthe scene to produce a multispectral three dimensional representation ofthe scene.

In certain embodiments, the visible light image and the infrared imageare combined to provide a combined multispectral image before beingoverlaid onto the three dimensional representation of the scene.

In certain embodiments, the apparatus further comprises imagesegmentation means arranged to identify objects located at differentdepths in the scene.

In certain embodiments, objects located at a depth in the scene greaterthan a first predetermined depth are removed from the representation ofthe scene.

In certain embodiments, the first predetermined depth is fixed.

In certain embodiments, the apparatus further comprising means toidentify an object of interest in the scene, wherein the firstpredetermined depth is based on the depth in the scene at which anidentified object of interest is located.

In certain embodiments, objects located at a depth in the scene smallerthan a second predetermined depth are removed from the representation ofthe scene.

In certain embodiments, the second predetermined depth is fixed.

In certain embodiments, the apparatus further comprises means toidentify an object of interest in the scene, wherein the secondpredetermined depth is based on the depth in the scene at which anidentified object of interest is located.

In certain embodiments, the overlaying means is arranged to selectivelyoverlay the infrared image onto objects in dependence on the depths atwhich the objects are located in the three dimensional representation ofthe scene.

In certain embodiments, the overlaying means is arranged to selectivelyoverlay the infrared image only onto objects located at predetermineddepths in the three dimensional representation of the scene.

In certain embodiments, the predetermined depths are fixed.

In certain embodiments, the apparatus further comprises means toidentify an object of interest in the scene, wherein the predetermineddepths are a range of depths extending to either side of the location ofan identified object of interest.

In certain embodiments, the range of depths are centred on the locationof an identified object of interest.

In certain embodiments, the apparatus further comprises means toidentify an object in the scene, wherein the identified object isremoved from the representation of the scene.

In certain embodiments, the at least one visible light imaging sensorcomprises at least two spaced apart imaging sensors arranged to form acyclopean camera arrangement; and

the apparatus further comprises an image combiner arranged to combineimages from the at least two spaced apart imaging sensors to provide afirst cyclopean image of a scene from a first virtual position.

In certain embodiments, the at least one infrared imaging sensorprovides a second image of the scene from a second position; and

the first virtual position and the second position of the at least oneinfrared imaging sensor are arranged to be substantially the same.

In certain embodiments, the at least one infrared imaging sensorcomprises an image sensor element, wherein the first cyclopean imagefrom the cyclopean camera arrangement appears to be generated on avirtual image sensor element within a virtual camera, and wherein theimage sensor element of the at least one infrared imaging sensor ispositioned at the spatial position of the virtual image sensor elementof the virtual camera.

In certain embodiments, the at least one infrared imaging sensor islocated between the two spaced apart imaging sensors.

In certain embodiments, the at least one infrared imaging sensor islocated at an equal distance from each of the two spaced apart imagingsensors.

In certain embodiments, the two spaced apart imaging sensors are housedwithin separate housings.

In certain embodiments, the at least one infrared imaging sensor isseparate from each of the housings for the two spaced apart imagingsensors.

In certain embodiments, the at least one infrared imaging sensor and thetwo spaced apart imaging sensors are all comprised within a commonhousing.

In certain embodiments, the representation generating means comprises aprocessor which compares the images from the at least two spaced apartimaging sensors and uses stereoscopic techniques to determine the depthsand/or distances

In certain embodiments, the representation generating means comprises astructured light scanner.

In certain embodiments, the at least one visible light imaging sensorviews structured light projected by the structured light scanner andprovides information regarding the viewed structured light to therepresentation generating means.

In certain embodiments, the visible light image and infrared image arepresented at substantially the same level of magnification when beingcombined to provide the combined multispectral image.

In certain embodiments, overlaying the infrared image onto the threedimensional representation of the scene comprises raycasting the visibleimage onto the three dimensional representation of the scene.

In certain embodiments, overlaying the visible light image onto thethree dimensional representation of the scene comprises raycasting thevisible light image onto the three dimensional representation of thescene.

In certain embodiments, overlaying the combined image onto the threedimensional representation of the scene comprises raycasting thecombined image onto the three dimensional representation of the scene.

In certain embodiments, the first image and second image are provided tothe second image combiner at substantially the same level ofmagnification.

In certain embodiments, the infrared image is at least one of: thethermal infrared wavelength range, the mid infrared wavelength range,the near infrared wavelength range, and the long infrared wavelengthrange.

In a fourth aspect the invention provides an imaging method, comprising:

generating a three dimensional representation of a scene usingrepresentation generating means;

obtaining an infrared image of the scene using at least one infraredimaging sensor; and

overlaying the infrared image onto the three dimensional representationof the scene to produce an infrared three dimensional representation ofthe scene using overlaying means.

In certain embodiments, the three dimensional representation of thescene is a three dimensional model.

In certain embodiments, the method further comprises obtaining a visiblelight image of the scene using at least one visible light imagingsensor; and

using the overlaying means to overlay the infrared image and the visiblelight image onto the three dimensional representation of the scene toproduce a multispectral three dimensional representation of the scene.

In certain embodiments, the visible light image and the infrared imageare combined to provide a combined multispectral image before beingoverlaid onto the three dimensional representation of the scene.

In certain embodiments, the method further comprises using imagesegmentation means to identify objects located at different depths inthe scene.

In certain embodiments, objects located at a depth in the scene greaterthan a first predetermined depth are removed from the representation ofthe scene.

In certain embodiments, the first predetermined depth is fixed.

In certain embodiments, the apparatus further comprises means toidentify an object of interest in the scene, wherein the firstpredetermined depth is based on the depth in the scene at which anidentified object of interest is located.

In certain embodiments, objects located at a depth in the scene smallerthan a second predetermined depth are removed from the representation ofthe scene.

In certain embodiments, the second predetermined depth is fixed.

In certain embodiments, the apparatus further comprises means toidentify an object of interest in the scene, wherein the secondpredetermined depth is based on the depth in the scene at which anidentified object of interest is located.

In certain embodiments, the overlaying means is arranged to selectivelyoverlay the infrared image onto objects in dependence on the depths atwhich the objects are located in the three dimensional representation ofthe scene.

In certain embodiments, the overlaying means is arranged to selectivelyoverlay the infrared image only onto objects located at predetermineddepths in the three dimensional representation of the scene.

In certain embodiments, the predetermined depths are fixed.

In certain embodiments, the apparatus further comprises means toidentify an object of interest in the scene, wherein the predetermineddepths are a range of depths extending to either side of the location ofan identified object of interest.

In certain embodiments, the range of depths are centred on the locationof an identified object of interest.

In certain embodiments, the apparatus further comprises means toidentify an object in the scene, wherein the identified object isremoved from the representation of the scene.

In certain embodiments, the at least one visible light imaging sensorcomprises at least two spaced apart imaging sensors arranged to form acyclopean camera arrangement; and

the method further comprises using an image combiner to combine imagesfrom the at least two spaced apart imaging sensors to provide a firstcyclopean image of a scene from a first virtual position.

In certain embodiments, the at least one infrared imaging sensorprovides a second image of the scene from a second position; and

the first virtual position and the second position of the at least oneinfrared imaging sensor are arranged to be substantially the same.

In certain embodiments, the at least one infrared imaging sensorcomprises an image sensor element, wherein the first cyclopean imagefrom the cyclopean camera arrangement appears to be generated on avirtual image sensor element within a virtual camera, and wherein theimage sensor element of the at least one infrared imaging sensor ispositioned at the spatial position of the virtual image sensor elementof the virtual camera.

In certain embodiments, the at least one infrared imaging sensor islocated between the two spaced apart imaging sensors.

In certain embodiments, the at least one infrared imaging sensor islocated at an equal distance from each of the two spaced apart imagingsensors.

In certain embodiments, the two spaced apart imaging sensors are housedwithin separate housings.

In certain embodiments, the at least one infrared imaging sensor isseparate from each of the housings for the two spaced apart imagingsensors.

In certain embodiments, the at least one infrared imaging sensor and thetwo spaced apart imaging sensors are all comprised within a commonhousing.

In certain embodiments, the representation generating means comprises aprocessor which compares the images from the at least two spaced apartimaging sensors and uses stereoscopic techniques to determine the depthsand/or distances

In certain embodiments, the representation generating means comprises astructured light scanner.

In certain embodiments, the at least one visible light imaging sensorviews structured light projected by the structured light scanner andprovides information regarding the viewed structured light to therepresentation generating means.

In certain embodiments, the visible light image and infrared image arepresented at substantially the same level of magnification when beingcombined to provide the combined multispectral image.

In certain embodiments, overlaying the infrared image onto the threedimensional representation of the scene comprises raycasting the visibleimage onto the three dimensional representation of the scene.

In certain embodiments, overlaying the visible light image onto thethree dimensional representation of the scene comprises raycasting thevisible light image onto the three dimensional representation of thescene.

In certain embodiments, overlaying the combined image onto the threedimensional representation of the scene comprises raycasting thecombined image onto the three dimensional representation of the scene.

In certain embodiments, the first image and second image are provided tothe second image combiner at substantially the same level ofmagnification.

In certain embodiments, the infrared image is at least one of: thethermal infrared wavelength range, the mid infrared wavelength range,the near infrared wavelength range, and the long infrared wavelengthrange.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is diagrammatically illustrated by way of example, in theaccompanying drawings in which:

FIG. 1 shows a schematic diagram of a scene being imaged by separatevisible and infrared cameras;

FIG. 2 shows a schematic diagram of the scene shown in FIG. 1 beingimaged by a virtual cyclopean camera arrangement;

FIG. 3 shows a schematic diagram of the scene shown in FIG. 1 beingimaged by a camera arrangement according to a first embodiment of thepresent invention;

FIG. 4 shows a flow diagram of a method of the first embodiment of thepresent invention;

FIG. 5 shows a schematic diagram of the scene shown in FIG. 1 beingimaged by a camera arrangement according to a second embodiment of thepresent invention;

FIG. 6 shows a flow diagram of a method of a second embodiment of thepresent invention; and

FIG. 7 shows a flow diagram of a further method of the second embodimentof the present invention.

DETAILED DESCRIPTION

FIG. 1 shows a schematic diagram of a scene being imaged by separatevisible light and infrared cameras, and is used here to exemplify theparallax problem.

As shown in FIG. 1, an object of interest 1 is present within a scenevisible in the fields of view of both a visible light camera 2 and aninfrared camera 3. The object of interest 1 is shown as a circle. Alsoin the scene visible to the two cameras 2 and 3, in the background thereare a number of background objects 4 a to 4 c, comprising a triangle 4a, a square 4 b and a pentagon 4 c. In the foreground there are a numberof foreground objects 5 a to 5 c, which are represented as a cross 5 a,a hexagon 5 b, and a star 5 c. From the perspective of the visible lightcamera 2 the object of interest 1 appears to be positioned between thesquare 4 b and the pentagon 4 c in the background, and also appears tobe positioned between the cross 5 a and the hexagon 5 b in theforeground. However from the perspective of the infrared camera 3 theobject of interest 1 appears to be positioned between the triangle 4 aand the square 4 b in the background and also appears to be positionedbetween the hexagon 5 b and the star 5 c in the foreground.

This represents an extreme case of the parallax problem, and in practiceit may be possible for the visible and infrared cameras 2 and 3 to bepositioned relatively close to one another, so that their differentviewpoints are close to each other, resulting in a viewed object sceneappearing similar even when viewed from the different viewpoints of thedifferent locations of the visible light and infrared cameras 2 and 3.However, even in this case, although the parallax problem may be reducedit still exists.

This parallax effect can cause noticeable problems when the images fromthe visible light camera 2 and the infrared camera 3 are combined, wherefor example the resultant combined image can be distorted leading to aneffect similar to the imaging system formed by the two cameras 2 and 3suffering from ‘double vision’. As a result of the images produced bythe two cameras being different because of parallax effects, if theimages from the two cameras are overlaid in correct alignment at aparticular object of interest in the image, this distortion effectbecomes increasingly large, and so increasingly apparent, for objects ofincreasing depth separation away from the depth position of theparticular object. That is, the distortion effect becomes increasinglylarge as the distance of the object from the cameras 2 and 3, becomesincreasing different from the distance of the object of interest, bothtoward objects (5 a, b, c) the cameras 2 and 3 and away (4 a, b, c) fromthe cameras 2 and 3.

In practice, where there is an object of interest visible, the imagesfrom the cameras 2 and 3 will commonly be overlaid in correct alignmentat the object of interest, so that the distortion effect will becomeincreasingly large and apparent for objects of increasing depthseparation away from the object of interest.

A parallax based problem similar to the above will also relate toindividual objects themselves when those objects have a significantdepth extending away and toward the cameras, where the images from thecameras 2 and 3 will be differently aligned for parts of an object atdifferent depth positions.

The problem of parallax disappears for objects at long distances, and inparticular at the long distances generally referred to in optics asbeing at infinity, but becomes an ever increasing problem as objectsmove closer to the two cameras. Corrective measures that are appliedgenerally need to know the distance to the object of interest. A problemis that in general any correction only properly applies for objects at aparticular distance from the cameras, and objects further away from orcloser to the cameras will have an incorrect correction applied.Similarly, such an incorrect correction will also be applied todifferent parts of the object of interest itself when it has asignificant depth within the scene.

FIG. 2 shows a schematic diagram of the scene shown in FIG. 1 beingimaged by a virtual cyclopean camera arrangement.

In the illustrated example of FIG. 2, a virtual cyclopean camera 6 isformed by combining the image outputs of two spaced apart cameras 2 and3 to provide a cyclopean image. By using appropriate processing theimages produced by the two cameras 2 and 3 can be combined to produce acyclopean image which appears to have been produced from the viewpointof the virtual cyclopean camera 6 that is situated between the twocameras 2 and 3, as shown in FIG. 2. The necessary processing requiredin any particular case can be readily carried out by the skilled personusing known techniques, and need not be discussed in detail. In oneexample, each camera 2 and 3 has a respective sensor.

Each camera 2 and 3 provides a slightly different view of the scene,where the view of the scene from each of the cameras 2 and 3 is detectedby the corresponding sensor in each camera 2 or 3. Through appropriateprocessing the images from the two sensors are combined to provide acyclopean image that appears as if it has been produced from the virtualcyclopean camera 6 located between the actual positions of the cameras 2and 3. The cyclopean image appears to have been generated on a virtualsensor within the virtual cyclopean camera 6. The position of thevirtual sensor of the virtual cyclopean camera 6 is situated midwaybetween the sensors of each of the two cameras 2 and 3 making up thevirtual cyclopean camera arrangement.

In other examples the position of the virtual sensor of the virtualcyclopean camera may be arranged to be at a location which is not midwaybetween the sensors of the two cameras making up the virtual cyclopeancamera arrangement. The location of the sensor of the virtual camera canbe selected to be any desired location, and the skilled person will beable to carry out the necessary image combination processing to do thiswithout difficulty. However, arranging the virtual sensor of the virtualcyclopean camera at a position midway between the sensors of the twocameras may provide advantages of simplifying the processing required tocombine the images from the two cameras into a virtual cyclopean imageand/or reducing problems due to objects visible in the image from onecamera not being visible in image from the other camera.

FIG. 3 shows a schematic diagram of the scene shown in FIG. 1 beingimaged by a multispectral camera arrangement 14 according to a firstembodiment of the present invention.

In the illustrated example of FIG. 3, the multispectral cameraarrangement 14 comprises a virtual cyclopean camera arrangement 13,which is formed by two spaced apart visible light imaging cameras 7 and8 and an image combiner 11. In the virtual cyclopean camera arrangement13, a virtual cyclopean visible light camera 9 is formed by combiningthe image outputs of two spaced apart visible light cameras 7 and 8operating over visible light bandwidths by the image combiner 11 toprovide a visible light cyclopean image. The two visible light imagesproduced by the two visible light cameras 7 and 8 are combined by theimage combiner 11 to produce a cyclopean visible light image whichappears to have been produced from the viewpoint of the virtual visiblelight cyclopean camera 9 that is situated centrally between the twospaced apart visible light cameras 7 and 8.

In one example, the visible light camera 7 has a visible light imagesensor element 7 a, and the visible light camera 8 has a visible lightimage sensor element 8 a. In one example the visible light sensorelements 7 a and 8 a of the respective visible light cameras 7 and 8each incorporate a two dimensional detector array, such as a CCD or CMOSchip. In other examples, other array technologies can be used.

Each visible light camera 7 and 8 provides a slightly different view ofthe scene, where the view of the scene from each of the visible lightcameras 7 and 8 is detected by the corresponding sensor in each visiblelight camera 7 or 8. The images from the two sensors are combined by theimage combiner 11 to provide a visible light cyclopean image thatappears as if it has been produced from the virtual cyclopean visiblelight camera 9 located between the actual positions of the visible lightcameras 7 and 8. The cyclopean visible light image appears to have beengenerated on a virtual visible light image sensor element 9 a within thevirtual visible light cyclopean camera 9. The position of the virtualvisible light sensor element 9 a of the virtual visible light cyclopeancamera 9 is situated between the visible light sensor elements 7 a and 8a of each of the two visible light cameras 7 and 8 making up the virtualcyclopean camera arrangement. The exact position of the virtual imageplane, equating to the position of the virtual sensor element 9 a of thevirtual cyclopean camera 9, can be readily determined by the personskilled in the art.

Referring to FIG. 3, the multispectral camera arrangement 14 furthercomprises an infrared (IR) imaging camera 10, which is positionedbetween the two visible light imaging cameras 7 and 8 at the same, orsubstantially the same, position as the virtual position of the virtualcyclopean visible light camera 9. The infrared camera 10 then sees thescene from the same, or substantially the same, viewpoint as theapparent virtual viewpoint of the virtual cyclopean visible light camera9. In some examples, the infrared camera 10 has an infrared image sensorelement 10 a incorporating a two dimensional detector array, such as aHgCdTe, Lead based, or Indium based, or bolometer material array. Otherarray technologies can be used. The infrared sensor element 10 a of theinfrared imaging camera 10 is positioned between the visible lightsensor elements 7 a and 8 a of the two visible light cameras 7 and 8 atthe same, or substantially the same, position as the virtual position ofthe virtual visible light sensor element 9 a of the virtual cyclopeanvisible light camera 9. Accordingly, the infrared sensor element 10 a ofthe infrared imaging camera 10 is positioned at, or close to, thevirtual imaging plane of the virtual cyclopean visible light camera 9.

In one example the two visible light cameras 7 and 8 forming the virtualcyclopean camera arrangement are in separate housings, thereby enablingthe infrared camera 10 to be positioned at the appropriate position.

The visible light cyclopean image produced from the image outputs of thetwo visible light cameras 7 and 8, which cyclopean image appears to befrom the virtual visible light cyclopean camera 9, and the infraredimage from the infrared camera 10 are combined by the image combiner 11to produce a combined infrared and visible light multispectral image.Since the infrared camera 10 is located at the same, or similar,position as the virtual position of the virtual cyclopean camera 9, anypossible parallax problems may be reduced or eliminated. Since theinfrared and visible light images which are being combined are, ineffect, being viewed from the same position in space, that is the actualview position of the infrared image and the apparent virtual viewposition of the cyclopean visible light image are the same position inspace, there is a reduced, or no, parallax problem in relation toobjects lying at different distances from the different cameras, andfurthermore there is a reduced, or no, parallax problem for differentparts of an object of interest.

FIG. 4 shows a flow diagram of a method according to the firstembodiment of the present invention, further describing the apparatusand method by which parallax may be corrected for combined visible andinfrared images of a scene.

The two visible light images from the two visible imaging light cameras7 and 8 forming the virtual visual light cyclopean camera arrangement14, referred to as the first visible light image 100 and the secondvisible light image 101 respectively, are combined by the image combiner11 in a visible light image combination step 102 to provide a cyclopeanvisual light image 103. As described above, this cyclopean visual lightimage 103 appears to have come from a virtual visible light cyclopeancamera 9 that has viewed the scene from a central position between theactual positions of the two visible light cameras 7 and 8.

Further, an infrared image 104 of the scene is obtained from the imaginginfrared camera 10. In the illustrated example the infrared camera 10operates over the thermal infrared wavelength range. In some examplesthe infrared camera 10 operates over the mid infrared wavelength range,to the extent that this definition differs to that of the thermalwavelength range. In some examples the infrared camera 10 operates overthe near infrared wavelength range. In some examples the infrared camera10 operates over the long infrared wavelength range.

The cyclopean visual light image 103 and infrared image 104 are thencombined by the image combiner 11 in an image combination step 105 toprovide a combined visible and infrared multispectral image 106. Theinfrared image information of the infrared image 104 is represented inthe combined multispectral image 106 by false colours corresponding todifferent temperatures, as is well known. In other examples other knownmethods of representing infrared image information may be used.

The combined visible light and infrared multispectral image 106 is thenpresented for display on a visual display unit (VDU) 12 in a displaystep 107. Instead of, or in addition to, being displayed the combinedvisible light and infrared image may be recorded for future display.

In the image combination step 105 the cyclopean visible light image 103and the infrared image 104 are combined at the same or substantially thesame magnification or zoom, such that when combined the features withinthe scene at visible light and infrared wavelengths correlate, so thatthe visible light and infrared parts of the combined image 106effectively overlie one another and are correctly aligned.

In some examples, to further facilitate the image combining processingundertaken by the image combiner, the respective detector making up thevisible light sensor elements 7 a and 8 a of the two visible lightcameras 7 and 8 and the infrared sensor element 10 a of the infraredcamera 10 can be arranged and chosen such that each detector comprises arespective detector array and the number of pixels of each detectorarray along an axis, or both orthogonal axes, of the detector array isthe same, or are related by an integer ratio, that is, are divisible oneby the other to provide an integer number.

In some examples the image combiner 11 carrying out the imagecombination may modify the visible and/or infrared images such that thenumber of pixels in each image in orthogonal image axes is the same, orare related by an integer ratio.

Normal image representation techniques can be used for the infrared partof the combined image. For example, false colours may be used in thecombined image 106 to represent the temperatures of objects within theviewed scene.

In examples where the images are digital images the image combiner maybe an image processor.

In the illustrated examples the combination of the two visible lightimages to form the cyclopean image and the combination of the cyclopeanimage with the infrared image are carried out by the same imageprocessor. In other examples separate dedicated image processors may beused to carry out the different combinations.

In the illustrated example the virtual cyclopean camera is a virtualvisible light cyclopean camera formed by two visible light cameras, andthe visible light cyclopean image is combined with an infrared imagefrom an infrared camera. In other examples this arrangement may bereversed so that the virtual cyclopean camera is a virtual infraredcyclopean camera formed by two infrared cameras, and the infraredcyclopean image is combined with a visible light image from a visiblelight camera.

In the illustrated example imaging cameras are used. In other examplesother types of imaging sensor may be used.

FIG. 5 shows a schematic diagram of the scene shown in FIG. 1 beingimaged by a multispectral camera arrangement 20 according to a secondembodiment of the present invention.

In the illustrated example of FIG. 5 the multispectral cameraarrangement 20 comprises a virtual cyclopean camera arrangement 13 andan infrared imaging camera 10 as described above regarding the firstembodiment, which are used to provide a combined infrared and visiblelight multispectral image.

The multispectral camera arrangement 20 further comprises depth ordistance sensing means, which in this examples is a structured lightscanner 21. The structured light scanner 21 projects a light pattern ofparallel bright and dark fringes and scans this projected light patternacross the scene viewed by the virtual cyclopean camera arrangement 13and the infrared imaging camera 10.

When the scanned fringes are projected onto objects or surfaces in theviewed scene the appearance of the fringes is distorted, with thedistortions depending on the locations of the objects or surfaces.Accordingly, by viewing the projected pattern of fringes and analysingtheir appearance it is possible to determine the locations in threedimensions of the objects or surfaces onto which the fringes areprojected. The principles of the use of structured light scanning toobtain three dimensional information are well known to the skilledperson and do not need to be discussed in detail herein.

The visible light cameras 7 and 8 of the virtual cyclopean cameraarrangement 13 are used to view the light pattern projected by thestructured light scanner 21 and the images from the visible lightcameras 7 and 8 are provided to a three dimensional modelling unit 22 inaddition to the image combiner 11.

The three dimensional modelling unit 22 uses the images of the projectedlight pattern received from the visible light cameras 7 and 8 togenerate a three dimensional model of the objects and surfaces in theviewed scene.

The three dimensional model from the three dimensional modelling unit22, the visible light cyclopean image 103 from the image combiner 11,and the infrared image 104 from the infrared camera 10 are supplied to araycaster unit 23. The raycaster unit 23 raycasts the visible lightcyclopean image 103 and the infrared image 104 onto the threedimensional model to overlay the visible light cyclopean image 103 andthe infrared image 104 onto the three dimensional model and produce athree dimensional model of the viewed scene augmented with both visiblelight and infrared information.

Using the two visible light cameras 7 and 8 forming the virtualcyclopean camera assembly 13 to view projected structured light patternis convenient when the virtual cyclopean camera assembly 13 is beingused because these two visible light cameras 7 and 8 are already presentso that the cost of additional cameras can be avoided. Further, usingthe images from these two cameras 7 and 8 to generate the threedimensional representation of the scene has the advantage that becausethe infrared camera 10 is at the position of the virtual centralcyclopean camera 9 as discussed above, the information on which thethree dimensional model is based will have been acquired from the sameviewpoint as that used to acquire both the cyclopean visible light image103 and the infrared image 104, so avoiding possible parallax orviewpoint problems.

FIG. 6 shows a flow diagram of a method according to the secondembodiment of the present invention, further describing an apparatus andmethod for three dimensional overlaying of visible and infrared imagesof a scene.

In the illustrated example of FIG. 6, a three dimensional representationof the scene is obtained, from which a three dimensional model ofobjects within the scene is produced.

In the illustrated example of FIG. 6 the first visible light image 100and the second visible light image 101 from respective ones of the twovisible imaging light cameras 7 and 8 forming the virtual visual lightcyclopean camera arrangement 13, are combined by the image combiner 11in a visible light image combination step 102 to provide a cyclopeanvisual light image 103. As described above, this cyclopean visual lightimage 103 appears to have come from a virtual visible light cyclopeancamera 9 that has viewed the scene from a central position between theactual positions of the two visible light cameras 7 and 8.

In the illustrated example of FIG. 6, in addition to being used to formthe cyclopean visible light image 103 the first and second visible lightimages 100 and 101 are supplied to the three dimensional modelling unit22 and used to determine three dimensional information of the scene. Thefirst and second visible light images 100 and 101 are used by the threedimensional modelling unit 22 to produce a depth map 208 of objects inthe scene. The three dimensional modelling unit 22 then uses the depthmap 208 to produce a three dimensional model 209 of the viewed scene.

As in the first embodiment, an infrared camera 10 is used to provide aninfrared image 104 of the scene. In one example this may be a thermalinfrared image of the scene. In the illustrated example the infraredcamera 10 operates over the thermal infrared wavelength range. In someexamples the infrared camera 10 operates over the mid infraredwavelength range, to the extent that this definition differs to that ofthe thermal wavelength range. In some examples the infrared camera 10operates over the near infrared wavelength range. In some examples theinfrared camera 10 operates over the long infrared wavelength range.

As shown in FIG. 6, the cyclopean visible light image 103, and thethermal infrared image 104 are overlaid, individually and/or afterhaving been combined, onto the three dimensional model 209 of the sceneby the raycaster unit 23 in a raycasting step 210.

During the raycasting step 210 the infrared image 104 is raycast ontothe three dimensional model 209. Thus, infrared texture can be createdon objects within the imaged scene. This can provide a three dimensionalmodel of the scene augmented with thermal information.

During the raycasting step 210, the cyclopean visible light image 103 israycast onto the three dimensional model 109. Thus, visible texture canbe created on objects within the imaged scene, which may be at anincreased level of resolution to that of the infrared texture. This canprovide a three dimensional model of the scene augmented with visibleinformation.

Thus, a combined visible light and infrared three dimensional augmentedmodel 211 of the scene can be provided augmented with both visible andthermal information. This augmented three dimensional model 211 can bepresented to a display such as a VDU 12 in a display step 212, and/ordisplayed on a three dimensional display in a display step 213.

According to this embodiment, the visible and infrared images can becombined within a three dimensional representation or model 211 of theimaged object scene, as opposed to a conventional two dimensionalrepresentation of the imaged object scene. The combined threedimensional visible and infrared augmented model 211 can then bemanipulated spatially, thereby enabling further information about theimaged scene to be determined.

In other examples, other methods could be used to combine the visibleand infrared images with the 3D model instead of raycasting.

In the illustrated example two cameras are used to view the projectedstructured light pattern. In other examples a different number ofcameras may be used. In some examples only one camera may be used.

In the illustrated example, the two cameras used to view the projectedstructured light pattern are the two cameras of the virtual cyclopeancamera that is used to generate the visible light image of the viewedscene. In other examples the structured light scanner and the virtualcyclopean camera may use different cameras.

In the illustrated example the depth or distance sensing means comprisesa structured light scanner which uses structured light information toobtain location information regarding objects and surfaces in the viewedscene.

In an alternative example, the depth or distance sensing means maycomprise a processor comparing the first and second visible light imagesand using stereoscopic techniques to determine the depth or distance ofobjects and surfaces in the viewed scene from the differences betweenthe first and second visible light images.

In other examples location information of the scene may be determined byother means, for example ultrasonic ranging, laser radar (IADAR/IIDAR)or standard RADAR, in place of the structured light scanning.

The illustrated example generates a depth map, and then uses the depthmap to produce a three dimensional model of the viewed scene. In otherexamples other methods of producing a three dimensional model which donot require the generation of a depth map may be used.

FIG. 7 shows a flow diagram of a further example of the presentinvention, where the possible spatial manipulation referred to abovewith reference to the previous example is undertaken.

As is discussed below infrared images may suffer from a limiteddisplayable contrast of dynamic range with respect to objects within thescene, and it may be advantageous to increase the proportion of thedisplayable dynamic range associated with images of objects of interest.Additionally, it can be useful to select objects within an imaged scenefor further analysis. Furthermore, the selection of objects may allowfor the removal of clutter within the imaged scene, for easiercomprehension of the imaged scene. The illustrated method of FIG. 7 mayfacilitate these aspects.

A scene being imaged both within the visible and/or infrared wavelengthrange may contain a range of objects but only a selection of theseimaged objects may be of interest.

Within the imaged scene there may be a range of objects of differentdetected temperatures. Some objects may be very hot, and/or some objectsmay be very cold, or there may be both very hot and very cold objectswithin the scene. The object or objects of interest within the imagedscene may have temperatures that only extend over a part of the totalrange of temperatures observable within the scene, for example if thehottest and/or coldest objects visible in the scene are backgroundobjects which are not actually objects of interest. This may result inthe display of temperature information regarding the objects of interestbeing obscured.

For example, if false colours are used to indicate temperatureinformation regarding objects within a viewed scene and the scene has atotal temperature range of 100K, with an available palette of 20 falsecolours to be used to represent these temperatures, then the temperaturerange represented by one false one colour would be approximately 5K.However, the objects of interest within the scene might only havetemperatures extending across a temperature range of only 10K so thatonly 2 or 3 colours will be used to represent these temperatures, sothat there is a limited effective dynamic range of displayabletemperatures for the objects of interest. In such a situation, theeffective displayable dynamic range of the infrared imaging camera withrespect to the objects of interest has been compromised, even though theinfrared camera itself may be able to resolve temperatures across theobjects of interest at a temperature resolution better than 5K or indeedsignificantly better than 5K.

As shown in FIG. 7 visible image data or visible light data 301,infrared data 302 and three dimensional or depth information 303 areprocessed by a processor using an image segmentation algorithm in animage segmentation step 304 to produce segmented object information 305in which objects located at different depths, that is distances from theimaging cameras, within the viewed scene are identified.

One form of manipulation of the three dimensional visible/infrared imageis to carry out image segmentation using depth information.

In a background removal step 306 objects located beyond a specifiedmaximum depth can be removed from the image. This maximum depth may befixed, or may be based on the depth of an object of interest identifiedin the image. For example, the maximum depth may be a fixed amount abovethe depth of an identified object of interest, or a fixed multiple ofthe depth of an identified object of interest. This removal ofbackground objects may reduce clutter and make the image more readilyunderstandable.

In other examples a foreground removal step may be carried out inaddition to, or as an alternative to, the background removal step. Inthe foreground removal step, objects located at less than a specifiedminimum depth can be removed from the image. This minimum depth may befixed, or may be based on the depth of an object of interest identifiedin the image. For example, the minimum depth may be a fixed amount lessthan the depth of an identified object of interest, provided this ismore than zero, or a fixed fraction of the depth of an identified objectof interest. This removal of foreground objects may reduce clutter andmake the image more readily understandable.

An object of interest can be selected within an image duringpost-processing, or an object of interest can be selected in real timethrough for example the imaging system (visible and/or infrared cameras)being focussed at the distance of the object of interest. Other means ofselecting objects of interest will be apparent to the person skilled inthe art, such as cursor or icon selection of objects or using a touchscreen to select objects of interest for example, or laser ranging.Other means of selecting objects of interest in real time will beapparent to the person skilled in the art.

In real time, the focusing arrangement of the camera(s) can be used todetermine a range of interest from the camera(s). This range of interestcan be the range at which the object or objects of interest are located.In other examples the range of interest may be a band of rangesextending to either side of the depth of the object or objects ofinterest. In one example the range of interest may be a band of rangescentred on the depth of the object or objects of interest.

In an image enhancement step 307 the segmented object information 305can be used to mask-off, hide, or remove objects in the scene that arenot at this range of interest from the cameras. In some examples theseobjects may not be displayed at all in the enhanced image. In otherexamples these objects may be displayed in the enhanced image as visiblelight images only without any thermal information.

Where false visible colours are used to represent temperature in theinfrared data 302, the infrared data 302 regarding the objects stillrepresented in the image enhanced scene may be provided with enhancedtemperature contrast. This may be possible where the masked off, hiddenor removed objects in the foreground or background were hotter or colderthan the still represented objects at the range of the object ofinterest The available palette of false colour used to representtemperature data can then be used to extend only over the smaller rangeof temperatures observable at the range of interest, where the object ofinterest is located, rather than the full range of temperatures in thefull imaged scene, where for example a hot object in the backgroundcould lead to reduced palette colour resolution over the object ofinterest.

After image enhancement and/or background removal the resulting imagemay be displayed on a display 212 or 213 in a display step 308.

The segmented object information 305 and/or the enhanced image dataproduced by the image enhancement step 307 may be used in furtherprocessing 309. Such further processing may, for example, includeprocessing for object tracking, automatic fault detection, and the like.

In some examples, objects can be selected in order that they are masked,removed, or hidden, with the remaining objects within the scene thenforming the objects of interest.

In some examples objects at a number of different ranges within theobject scene can be selected in order that they are masked, removed, orhidden.

During post-processing a specific object can be selected to furtherenhance contrast with respect to other objects at that range and allowenhancement algorithms such as high dynamic range algorithms to be used.Similarly, in real time such specific image selection and imagingprocessing could be provided for.

In some examples, three dimensional or depth information and infraredimage data are processed. In other examples, three dimensional or depthinformation and combined visible light and infrared image data areprocessed.

In the illustrated examples two visible light images from two camerasare used to form the cyclopean image. In other examples three or morevisible light images from three or more cameras are used to form thecyclopean image.

In some examples the cyclopean camera operates with two infrared camerasthat are used to obtain a cyclopean infrared image. In some examples thecyclopean camera operates over the thermal infrared wavelength range. Insome examples the cyclopean camera operates over the mid infraredwavelength range, to the extent that this definition differs to that ofthe thermal wavelength range. In some examples the cyclopean cameraoperates over the near infrared wavelength range. In some examples thecyclopean camera operates over the long infrared wavelength range.

In some examples, a camera operating over visible wavelengths ispositioned between the two infrared cameras of the infrared cyclopeancamera.

In some examples the visible imaging cameras forming the cyclopeancamera are contained within a single housing. In some examples theinfrared imaging camera is also contained within the same housing.

In some examples the image combiner 11 carrying out the imagecombination may modify the visible and/or infrared images such that thenumber of pixels in each image in orthogonal image axes is the same, orare related by an integer ratio.

Normal image representation techniques can be used for the infrared partof the combined image. For example, false colours may be used in thecombined image 106 to represent the temperatures of objects within theviewed scene.

In examples where the images are digital images the image combiner maybe an image processor.

In the illustrated example of the second embodiment the visible lightimages from the visible light cameras are supplied directly to the threedimensional modelling unit, In other examples these may be suppliedthrough the image combiner 11.

The features disclosed in the different embodiments discussed above andthe alternatives are not exclusive. Therefore, features in any or all ofthe embodiments and alternatives may be exchanged and/or combined ifdesired.

The apparatus described above may be implemented at least in part insoftware. Those skilled in the art will appreciate that the apparatusdescribed above may be implemented using general purpose computerequipment or using bespoke equipment. The hardware elements, operatingsystems and programming languages of such computers are conventional innature, and it is presumed that those skilled in the art are adequatelyfamiliar therewith.

Those skilled in the art will appreciate that while the foregoing hasdescribed what are considered to be the best mode and, whereappropriate, other modes of performing the invention, the inventionshould not be limited to specific apparatus configurations or methodsteps disclosed in this description of the preferred embodiment. It isunderstood that various modifications may be made therein and that thesubject matter disclosed herein may be implemented in various forms andexamples, and that the teachings may be applied in numerousapplications, only some of which have been described herein. It isintended by the following claims to claim any and all applications,modifications and variations that fall within the true scope of thepresent teachings.

Those skilled in the art will recognize that the invention has a broadrange of applications, and that the embodiments may take a wide range ofmodifications without departing from the inventive concept as defined inthe appended claims.

The invention claimed is:
 1. An apparatus for multispectral imaging,comprising: a cyclopean camera arrangement comprising at least twospaced apart imaging sensors operating at a first waveband; an imagecombiner arranged to combine images from the at least two spaced apartimaging sensors to provide a first cyclopean image of a scene from afirst virtual position; a further imaging sensor operating at a secondwaveband different from the first waveband and arranged to provide asecond image of the scene from a second position; image combining meansarranged to combine the first cyclopean image and second image of thescene to provide a multispectral image; means for selecting objects ofinterest within the scene; and depth sensing means configured todetermine a depth or distance of objects and surfaces in the scene fromdifferences between the images from the at least two spaced apartimaging sensors operating at the first waveband; wherein the firstvirtual position of the cyclopean camera arrangement and the secondposition of the further sensor are arranged to be substantially thesame; and wherein either the first waveband is infrared and the firstcyclopean image is represented via a false color palette or the secondwaveband is infrared and the second image is represented via a falsecolor palette; and the false color palette used to represent either thefirst cyclopean image or the second image extends only over the range oftemperatures observable within a range of interest, the range ofinterest corresponding to depths between a specified minimum depth and aspecified maximum depth.
 2. The apparatus according to claim 1, whereinthe second image combiner is the image combiner.
 3. The apparatusaccording to claim 1, wherein the first waveband is visible light andthe second waveband is infrared.
 4. The apparatus according to claim 1,wherein the first waveband is infrared and the second waveband isvisible light.
 5. The apparatus according to claim 1, wherein thefurther imaging sensor comprises an image sensor element, wherein thefirst image from the cyclopean camera arrangement appears to begenerated on a virtual image sensor element within a virtual camera,wherein the image sensor element of the further imaging sensor ispositioned at the spatial position of the virtual image sensor elementof the virtual camera.
 6. The apparatus according to claim 1, whereinthe further imaging sensor is located between the two spaced apartimaging sensors.
 7. The apparatus according to claim 6, wherein thefurther imaging sensor is located at an equal distance from each of thetwo spaced apart imaging sensors.
 8. The apparatus according to claim 1,wherein the cyclopean camera arrangement comprises two spaced apartimaging sensors housed within separate housings.
 9. The apparatusaccording to claim 8, wherein the further imaging sensor is separatefrom each of the housings for the two spaced apart imaging sensors. 10.The apparatus according to claim 1, wherein the further imaging sensorand the two spaced apart imaging sensors are all comprised within acommon housing.
 11. The apparatus according to claim 1, wherein thefirst image and second image are provided to the second image combinerat substantially the same level of magnification.
 12. The apparatusaccording to claim 3, wherein the second waveband is at least one of:the thermal infrared wavelength range, the mid infrared wavelengthrange, the near infrared wavelength range, and the long infraredwavelength range.
 13. The apparatus according to claim 4, wherein thefirst waveband is at least one of: the thermal infrared wavelengthrange, the mid infrared wavelength range, the near infrared wavelengthrange, and the long infrared wavelength range.
 14. The apparatusaccording to claim 1, wherein the at least two spaced apart imagingsensors are cameras and/or the further imaging sensor is a camera.
 15. Amethod for multispectral imaging, comprising: obtaining a firstcyclopean image of a scene from a first virtual position using acyclopean camera arrangement comprising at least two spaced apartimaging sensors operating at a first waveband by combining images fromthe at least two spaced apart imaging sensors to provide the firstcyclopean image of the scene; obtaining a second image of the scene froma second position using a further imaging sensor operating at a secondwaveband different from the first waveband, and wherein either the firstwaveband is an infrared waveband and the first cyclopean image isrepresented via a false color palette or the second waveband is aninfrared waveband and the second image is represented via a false colorpalette; combining the first cyclopean image and the second image of thescene to provide a multispectral image; selecting an object of interestwithin the scene; determining a depth or distance of objects andsurfaces in the scene from differences between the images from the atleast two spaced apart imaging sensors operating at the first waveband;and adjusting the false color palette used to represent either the firstcyclopean image or the second image in the multispectral image to extendonly over the range of temperatures observable within a range ofinterest, the range of interest corresponding to depths between aspecified minimum depth and a specified maximum depth; wherein the firstvirtual position of the cyclopean camera arrangement and the secondposition of the further sensor are arranged to be substantially thesame.
 16. The apparatus according to claim 15, wherein the firstwaveband is visible light and the second waveband is infrared.
 17. Theapparatus according to claim 15, wherein the first waveband is infraredand the second waveband is visible light.
 18. The method according toclaim 15, wherein the further imaging sensor comprises an image sensorelement, wherein the first image from the cyclopean camera arrangementappears to be generated on a virtual image sensor element within avirtual camera, wherein the image sensor element of the further imagingsensor is positioned at the spatial position of the virtual image sensorelement of the virtual camera.
 19. The method according to claim 15,wherein the further imaging sensor is located between the two spacedapart imaging sensors.
 20. The method according to claim 19, wherein thefurther imaging sensor is located at an equal distance from each of thetwo spaced apart imaging sensors.
 21. The method according to claim 15,wherein the cyclopean camera arrangement comprises two spaced apartimaging sensors housed within separate housings.
 22. The methodaccording to claim 21, wherein the further imaging sensor is separatefrom each of the housings for the two spaced apart imaging sensors. 23.The method according to claim 15, wherein the further imaging sensor andthe two spaced apart imaging sensors are all comprised within a commonhousing.
 24. The method according to claim 15, wherein the first imageand second image are provided to the second image combiner atsubstantially the same level of magnification.
 25. The method accordingto claim 16, wherein the second waveband is at least one of: the thermalinfrared wavelength range, the mid infrared wavelength range, the nearinfrared wavelength range, and the long infrared wavelength range. 26.The method according to claim 17, wherein the first waveband is at leastone of: the thermal infrared wavelength range, the mid infraredwavelength range, the near infrared wavelength range, and the longinfrared wavelength range.
 27. The method according to claim 15, whereinthe at least two spaced apart imaging sensors are cameras and/or thefurther imaging sensor is a camera.